Repository navigation
Delphi shadow: a job graph that cannot finish no longer holds up other conversations' full-pipeline runs; optional same-host graph-stage daemon, off by default - #3447
Merged
Conversation
…d holds only its own conversation When a graph's attempt cannot be proven finished (its scope stays held), the scheduler no longer waits on it forever. After the job deadline the graph moves to a deferred list that is polled one per cycle, and the full-pipeline shadow keeps admitting every other conversation; only the conversation whose graph scope is held is skipped. The environments the graph path has used are recorded in the state file before any admission, so a graph admitted under a list that was later emptied is still found and recovered after a restart. The job daemon can bind its retained journal to one host identity file (POLIS_JOBS_JOURNAL_HOST_FILE): a journal written under another host is refused at start-up rather than read as exit evidence. Adds Rust tests for a lost attempt after a restart with the list emptied and for a committed admission whose reply was lost, and two Docker process checks for recovery and for the scheduler under its deployed memory and CPU limits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
…ournal volume and deploy check Adds the delphi-shadow-graph Compose service (profile delphi-shadow-graph, off unless DELPHI_SHADOW_GRAPH_ENABLED=1): the job daemon restricted to the four graph stages at concurrency one, with its own journal volume, a read-only host identity file bound in for the journal's host binding, and its own memory and CPU ceilings (6 GiB and 2 CPUs by default). The daemon refuses to start unless it runs as PID 1 of its own container with exactly those stages, concurrency one and a host identity file set. The deploy hook starts it only when the full-pipeline shadow is selected and the setting is on, and first checks the rendered Compose settings with scripts/check-shadow-graph.py (names only); if they are wrong, neither shadow is started. With the setting off, nothing in the deploy changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
… retried a bounded number of times; refusals are reconsidered by build A finished graph's publication is now confirmed by reading the served bundle and checking that its root artifact is this graph's leaf at the next generation, not by the publish call's reply alone. A publish whose commit is durable but whose reply was lost is recognised as published; a transient database error before commit (connection, serialization, deadlock, lock or too-many-connections classes) is retried at most three times, each on a fresh connection, and then counted failed. A definite refusal is not retried. If the served bundle cannot be read, the graph stays in flight and is resolved on a later poll. A refused conversation's recorded observation now carries the admission rule and the build's code version, so a later build looks at it again once. A graph admission that errored is followed by a recovery read, so an admission that committed before its reply was lost is picked up before any full-pipeline run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
… the recovery runbook is documented A graph waiting past its deadline now logs the reason it is still held (nodes still active, a process exit not yet confirmed, or reconciliation pending) and the count of attempts with no confirmed exit, instead of one fixed reason. A job daemon with a host-bound journal logs one line per retained journal entry saying whether the attempt's exit was proven and why. docs/configuration.md gains a section on the same-host graph executor: its profile, host identity file, journal volume, deadlines, memory ceilings and what to drain before a rollback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
…st container The graph executor's validator test located scripts/check-shadow-graph.py by its own path, which inside the Delphi test container resolves to /scripts and does not exist, so every shard of the Delphi Python tests stopped with a collection error. It now reads the validator from the checkout the other Compose tests read ($POLIS_CHECKOUT_DIR), and the deploy-hook runner copies the validator there with its other inputs. The test that the shadow yields CPU and memory to every other service failed once the graph executor was added, because the executor carries the shadow's own lowest weight. Both shadow daemons must now carry exactly that weight and OOM adjustment, and every other service must still outrank them; no assertion about any other service changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
Brings in the typed graph step and model registry. Three files conflicted only because both sides appended at the same end: docs/configuration.md, example.env and queue-rs/tests/delphi_shadow.rs. Each keeps edge's appended text first, unchanged, followed by this branch's appended text, unchanged. docker-compose.yml, queue-rs/src/shadow/graph.rs and scripts/after_install.sh merged without conflict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
…ing model setting as the shadow daemon does DELPHI_EMBED_MODEL_BY_ENV is now passed through to delphi-shadow-graph the way the registry passes it to delphi-shadow and the scheduler: listed by name, with no default of its own, so with the setting unset (today) it is absent from the container and nothing changes. Each graph's model is still decided and sealed by the scheduler at admission, and graph children's environment stays the existing allowlist. The deploy check refuses a graph executor whose setting differs from the scheduler's; tests cover the check and that both daemons list the setting the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
…de directly delphi/tests/shadow/graph_recovery.py admits one test-model node through the SQL admission function so that a test-build daemon has a real attempt to lose and recover, as the graph proofs already do. The check that every product admission goes through the shared entry now lists it beside those proofs. The proof's description says it needs a test-build image, since a production daemon refuses to run a test-only model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs
Contributor
Delphi Coverage Report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Observable changes
None with today's settings.
DELPHI_SHADOW_GRAPH_CONVERSATIONSis empty andDELPHI_SHADOW_GRAPH_ENABLEDis not set, so the new code makes no query, opens no connection and writes no log line, the scheduler's state file is byte-identical, and the deploy hook skips the new block. Rendered withdocker compose configunder every profile with production-shaped settings, this branch differs from edge only by the new servicedelphi-shadow-graph(its own profile, off) and its volumedelphi-shadow-graph-state; networks are identical, and the hook's existing shadow settings check names nothing.An operator who turned them on would see:
DELPHI_SHADOW_GRAPH_ENABLED=1(locally: thedelphi-shadow-graphprofile besidedelphi-shadow): adelphi-shadow-graphcontainer that claims the four graph steps (graph_embed,graph_cluster,graph_topics,graph_narrative) in the graph environment, one step at a time, as PID 1 of its own namespace, offline, with topic modelnoneand no provider key. Defaults: 6 GiB (DELPHI_SHADOW_GRAPH_CONTAINER_MEMORY), 2 CPUs (DELPHI_SHADOW_GRAPH_CONTAINER_CPUS), the same lowest CPU weight and highest OOM score as the full-pipeline shadow. Its attempt journal lives on the named volume, bound to the host by/etc/machine-id(or the file named inDELPHI_SHADOW_GRAPH_HOST_ID_FILE); a journal bound to another host, or holding entries written before the binding existed, is refused (journal_host_mismatch,journal_host_unrecorded), and a second daemon cannot take the journal's lock. It receivesDELPHI_EMBED_MODEL_BY_ENVexactly as the scheduler and the full-pipeline shadow do. At deploy,scripts/check-shadow-graph.pyvalidates the rendered settings and prints names only, never values.DELPHI_SHADOW_JOB_DEADLINE_SECONDS(default 14,400) is cancelled through the existing fenced cancel, logged once asgraph_recovery_pending, and followed in the background. Other conversations' full-pipeline runs resume. That conversation stays out of the full-pipeline shadow while its graph scope is held, and no new graph is admitted while one is deferred. Publication is accepted only when the served root is this graph's own, read back after every outcome. A publish that fails with a transient database error before commit is tried up to three times in all (graph_publish_retry), then counted failed (graph_publish_failed); a served read that itself fails keeps the graph to be resolved on a later poll. A conversation refused admission is looked at again once per build, so every deploy re-evaluates each listed conversation once.No schema change
No migration, no SQL function added or replaced, no table or role touched. No Node handler and no Rust API route changed.
What it does
The Delphi shadow is a background scheduler that runs the Delphi pipeline on chosen conversations into shadow result environments that readers never serve. It can also run a listed conversation as a four-node job graph in a separate shadow environment. That path was already in edge but switched off. This branch fixes two failure modes in it and adds a way to run its steps on the same host:
docs/configuration.md("Same-host graph executor and recovery").Test harness: the deploy-hook test runner now copies
scripts/check-shadow-graph.py, so the graph executor's Compose tests run in the Delphi test container. The repository check that every graph admission goes through the shared entry now listsdelphi/tests/shadow/graph_recovery.pybeside the other graph proofs, because that proof admits one test-model node directly through SQL. That proof needs the test build of the Delphi image.The vote, comment and next-comment paths
Nothing here is on them. No Node or Rust API code changed, and there is no migration. With the list empty, the new code makes no query at all. Once it is set, the scheduler's new database work is read-only, on its existing read connection, one connection at a time,
REPEATABLE READ, withstatement_timeoutandlock_timeoutof 1 s, against queue and graph tables only. The graph daemon writes only through the existing fenced queue functions, under its own queue login. No connection pool is shared with votes, comments or the next comment, and no lock is taken onconversations,votes,commentsorparticipants. The graph daemon shares the full-pipeline shadow's CPU weight and OOM score, which every other service outranks, and runs within its own memory and CPU ceilings: it yields first, though that reserves no capacity for participation. No vote is read or written. The vote-sign lint reports 0 new sites, and the recovery proof asserts the votes table is empty. The Delphi shadow integration tests inqueue-rs/tests/delphi_shadow.rs, which include the tests that the scheduler's sessions never make a vote, a comment or the next comment wait, pass.Checks
At this branch's head, on an arm64 Docker host that other work was also using (load 10 to 47). Counts overlap across suites.
make check-vote-sign-lintmake check-lintpython3 ci/check_graph_registry.pyand its unittestmake check-queue-rs(includes the Delphi shadow scheduler and daemon integration tests)delphi_shadow20,jobs_integration27); Clippy in both feature setsmake check-job-graphsjobs::journal::tests::one_daemon_per_journal_directory("free again once released",queue-rs/src/jobs/journal.rs:236). Re-run alone: pass, Rust 303 passed, 1 ignored; Jest 48; 18 proof recordsmake check-polis-cli--target finaland--target testdelphi-shadow-graphand its volume differ. The check accepts the graph on with defaults and with a list. It refuses the graph environment equal to the shadow's or to a served one, CPUs 0, the profile not rendered, and production's settings with the graph offmake check-embeddings-tableCOMPARE OKmake check-dynamo-writersPASSdelphi/tests/shadow/graph_recovery.py(test image; PostgreSQL 17 with this release's migrations)PASS: container replaced, same container restarted, journal lost (second daemon refused in each; the exact attempt recovered in the first two; in the third, nothing invented exit proof or released the scope); no votesdelphi/tests/shadow/graph_scheduler_limits.py(final image, 256 MiB and 0.25 CPU)delphi/tests/shadow/rehearsal.sh, unchangeddocker run … 2>&1 | tail -1, line 119) took a Hugging Face retry warning as its last line instead ofrefused:OSError. Run alone, the same probe on the same image answersrefused:OSError. Re-run: pass. 18 of 18 shadow jobs succeeded; granite embeddings 1,208 rows in 18 sets; the served pointer is unchanged; internet blocked at start and end; image and by-name vectors identical over 441 texts; nograph_log line from the schedulermake check-delphi-python-teststests/poller/test_promotion_postgres.pyfailingset_payload_type's assertion (tests/poller/test_backfill_postgres.py:55,[('json',), ('jsonb',)]): the shared-table race between those tests'jsonandjsonbcases, already red on edge and outside this change. 5,064 tests collectedKnown limits
The graph list is empty in production and the graph daemon's profile is off. Each of these is a condition to meet before the list is ever set:
/etc/machine-id, passes the binding. A different boot then counts as exit proof for an attempt that may still be running. Writes stay safe: finalize is fenced by owner, attempt and epoch and needs that attempt's exit proof, each node has one terminal attempt, and graph children hold no database setting. The harm is a false record and two machines computing the same node. Needed: an entry from another boot is confirmed only after the attempt's lease has expired by database time.queue-rs/tests/delphi_shadow.rsfor at least the lost admission reply, the emptied list with no flight in state, and the hold while a graph is deferred.graph_recovery.pyandgraph_scheduler_limits.pyare run by hand, and the limits proof cannot reuse its output directory. Needed: onemaketarget that builds the images, applies the release to an owned PostgreSQL and runs both, plus a row indocs/local-checks.md.DELPHI_SHADOW_CONTAINER_MEMORY, is not indocs/configuration.mdorexample.env, and macOS Docker Desktop has no/etc/machine-id. Needed: a short section naming both memory settings, a measured pair that works, and that the graph daemon is optional.🤖 Generated with Claude Code
https://claude.ai/code/session_01KvftUAdasWzxmMYdioZJWs